{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Populating the interactive namespace from numpy and matplotlib\n"
     ]
    }
   ],
   "source": [
    "# Plot parameters\n",
    "sns.set()\n",
    "%pylab inline\n",
    "pylab.rcParams['figure.figsize'] = (4, 4)\n",
    "plt.rcParams['xtick.major.size'] = 0\n",
    "plt.rcParams['ytick.major.size'] = 0\n",
    "# Avoid inaccurate floating values (for inverse matrices in dot product for instance)\n",
    "# See https://stackoverflow.com/questions/24537791/numpy-matrix-inversion-rounding-errors\n",
    "np.set_printoptions(suppress=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>\n",
       ".pquote {\n",
       "  text-align: left;\n",
       "  margin: 40px 0 40px auto;\n",
       "  width: 70%;\n",
       "  font-size: 1.5em;\n",
       "  font-style: italic;\n",
       "  display: block;\n",
       "  line-height: 1.3em;\n",
       "  color: #5a75a7;\n",
       "  font-weight: 600;\n",
       "  border-left: 5px solid rgba(90, 117, 167, .1);\n",
       "  padding-left: 6px;\n",
       "}\n",
       ".notes {\n",
       "  font-style: italic;\n",
       "  display: block;\n",
       "  margin: 40px 10%;\n",
       "}\n",
       "img + em {\n",
       "  text-align: center;\n",
       "  display: block;\n",
       "  color: gray;\n",
       "  font-size: 0.9em;\n",
       "  font-weight: 600;\n",
       "}\n",
       "</style>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%html\n",
    "<style>\n",
    ".pquote {\n",
    "  text-align: left;\n",
    "  margin: 40px 0 40px auto;\n",
    "  width: 70%;\n",
    "  font-size: 1.5em;\n",
    "  font-style: italic;\n",
    "  display: block;\n",
    "  line-height: 1.3em;\n",
    "  color: #5a75a7;\n",
    "  font-weight: 600;\n",
    "  border-left: 5px solid rgba(90, 117, 167, .1);\n",
    "  padding-left: 6px;\n",
    "}\n",
    ".notes {\n",
    "  font-style: italic;\n",
    "  display: block;\n",
    "  margin: 40px 10%;\n",
    "}\n",
    "img + em {\n",
    "  text-align: center;\n",
    "  display: block;\n",
    "  color: gray;\n",
    "  font-size: 0.9em;\n",
    "  font-weight: 600;\n",
    "}\n",
    "</style>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\newcommand\\bs[1]{\\boldsymbol{#1}}\n",
    "\\newcommand\\norm[1]{\\left\\lVert#1\\right\\rVert}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span class='notes'>\n",
    "    This content is part of a series following the chapter 2 on linear algebra from the [Deep Learning Book](http://www.deeplearningbook.org/) by Goodfellow, I., Bengio, Y., and Courville, A. (2016). It aims to provide intuitions/drawings/python code on mathematical theories and is constructed as my understanding of these concepts. You can check the syllabus in the [introduction post](https://hadrienj.github.io/posts/Deep-Learning-Book-Series-Introduction/).\n",
    "</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "\n",
    "The [2.4](https://hadrienj.github.io/posts/Deep-Learning-Book-Series-2.4-Linear-Dependence-and-Span/) was heavy but this one is light. We will however see an important concept for machine learning and deep learning. The norm is what is generally used to evaluate the error of a model. For instance it is used to calculate the error between the output of a neural network and what is expected (the actual label or value). You can think of the norm as the length of a vector. It is a function that maps a vector to a positive value. Different functions can be used and we will see few examples.\n",
    "\n",
    "\n",
    "# 2.5 Norms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Norms are any functions that are characterized by the following properties:\n",
    "\n",
    "1- Norms are non-negative values. If you think of the norms as a length, you easily see why it can't be negative.\n",
    "\n",
    "2- Norms are $0$ if and only if the vector is a zero vector\n",
    "\n",
    "3- Norms respect the triangle inequity. See bellow.\n",
    "\n",
    "4- $\\norm{\\bs{k}\\cdot \\bs{u}}=| \\bs{k}| \\cdot \\norm{\\bs{u}}$. The norm of a vector multiplied by a scalar is equal to the absolute value of this scalar multiplied by the norm of the vector.\n",
    "\n",
    "It is usually written with two horizontal bars: $\\norm{\\bs{x}}$\n",
    "\n",
    "# The triangle inequity\n",
    "\n",
    "The norm of the sum of some vectors is less than or equal to the sum of the norms of these vectors.\n",
    "\n",
    "$$\n",
    "\\norm{\\bs{u}+\\bs{v}} \\leq \\norm{\\bs{u}}+\\norm{\\bs{v}}\n",
    "$$\n",
    "\n",
    "### Example 1.\n",
    "\n",
    "$$\n",
    "\\bs{u}=\n",
    "\\begin{bmatrix}\n",
    "    1 & 6\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "and\n",
    "\n",
    "$$\n",
    "\\bs{v}=\n",
    "\\begin{bmatrix}\n",
    "    4 & 2\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\norm{\\bs{u}+\\bs{v}} = \\sqrt{(1+4)^2+(6+2)^2} = \\sqrt{89} \\approx 9.43\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\norm{\\bs{u}}+\\norm{\\bs{v}} = \\sqrt{1^2+6^2}+\\sqrt{4^2+2^2} = \\sqrt{37}+\\sqrt{20} \\approx 10.55\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check these results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 6])"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "u = np.array([1, 6])\n",
    "u"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([4, 2])"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v = np.array([4, 2])\n",
    "v"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([5, 8])"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "u+v"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9.4339811320566032"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.linalg.norm(u+v)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10.554898485297798"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.linalg.norm(u)+np.linalg.norm(v)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQIAAAD9CAYAAABX/HkoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd0XPWd9/H3jGY06sXSqLvgws82BpteQmiBUEII4RAg\nwBJ6aG5k9znJ5mR3z7MnefY8uw/FmBIIJmBKCIQewBhCIJhiDKYY29eSsY016m1UZzSje58/RpYt\nozq6ZTT6vs7hYI009/sdafTR/d3f797rMgwDIcTU5na6ASGE8yQIhBASBEIICQIhBBIEQggkCIQQ\ngMeuQtFon9Ha2m1XOfLzM7Cr3tq1f8DtdnHFFdfZUs/O1yb1Jn89vz/bNdrX2LZH4PGk2FXKkXp2\nSvbvpdSznwwNhBASBEIICQIhBBIEQggkCIQQSBAIIZAgEEIgQSCEQIJACIEEgRACCQIhBBIEQggk\nCMQksSv4DTWddU63kbQkCIQtntnxIh29naM+NpyoHuEvlS+jG/qIzx3PNsV+cV2PQCnlAR4FZgFR\n4AZN03aY2JdIMqFomIgeGfGxXcFv+JP2HF5PCtFo36Cv7dWjNPY0sbVZY1HhgiG3N1wdMbp4L0xy\nHpCiadp3lFJnAr8DLjavLTEVHZI7g18dtwK/P5vGxo6Bx0PRMPd+/gcuOfQ65k+b52CHySveINgB\neJRSLiAX6DWvJSEGMzC49NAfU5Fd5nQrSSveIOgEDgG2AwXA+aZ1JMRB0j1pSRECET1KKNKDn2yn\nW/mWeINgJfC6pmm/VkqVA28rpRZpmjbinoHfb+83wK56brfL1np21zKjXlqal4JpWRRmZo/42Fjr\nDffckbY5ErO+n1/Wb+f1yr/TEwnRHekZ+K8nEqKsw80lkbnM/vlS239+o4k3CFqAfUdk2vq3M+qF\n2A4c91nt4HGmlXTdwO122VbPztdmVr1QKEJzSydGt3fEx+q7G/moaSOB1gaOKzmSo4uXAPBB7SZe\n2vka/37Cv5DmSRvyucNt047XN7AtSgh2d7GjtWrgsZQ+g3OqUpm7uQb/sksA+38XRhNvENwFrFFK\nvQt4gV9pmtYT57aEAEA3dN765l2WnnwVz25ex7o9bw8EAYZBR28ndd0NzMqZ4WyjQ6juqOGjuk/Y\nVP8Z7b37f8nLGyJcsLmP1OZGUkvLyDjscAe7HF5cQaBpWhdwqcm9iCluW0slhxcuwO1ys7VFoyjD\nP/C5E8uOZUPNRrK8mQ52OFhbOMjHdZvZWPcpNV2xxU4FadM4d9b32LTnI476uIn5O9oHvj7vrO/j\nco16ZXFH2HZfAyFGU5FVSnZqFi3dbWxvqeTnh/9s0OenZ5dTmF7gUHcxoWiYzxu3sLHuU7TWKgwM\n0j3pfKfseI4rOYo5ubPQe3oofPINpu3eHwIpWdnknHCSg52PTIJAJIxcXw4A7+zeQHpKGgsL1KDP\nZ3jTnWgL3dDZ3lLJxrpP+bxxC716hBRXCocXLuS4kqNYVDAfb8r+YxLu9HSOXv7vVP/uP4k0NgKQ\ne/oZuFNTHel/LCQIRMLZ0rCdefmzcbv2r4Df3lLJITYfGxhq3D8rZwbHlRzF0UWLyUodephiRKPU\n3beaSGMj+WefQ9vf3iLvtDPsbH3cJAhEwmnpCbIwb/6gx75q3s5Fc61frjLSuP/YkqMoPuC4xVAM\nw6D+jw/TU7mDvDPPwv+Ty0g7ZDae3FzLe58ICQKRcGbmlhPs3T++fr/mY44oXGjZgbZQJMRHtZ+M\nOO4fa+3ml16g46MPyVy8BP8lPwUg+5jjLOnbTBIEIuH87MifcP8Hj/O09jwet4e5ebOZlz/H1BoH\njvu/aPqKcF/viOP+sWh/fwMtL7+Ib8ZMSm+4CZd78pzcK0EgEk5+ei43HH6VJdseatw/b9osjixc\nMuK4fzTd27dR9+gaPPnTKF+2AndampltW06CQCS90cb9i2bOntBKv97aGmruuweXx0v5spV48vLN\nat02EgQiKY1lvt+MYw7RjnYCq+5ED4UoX7YC3/TpJnRvPwkCkTR0Q6ctHOSZHS+xvWXHqPP9E64X\n6aVm9SoijY0UXXEVmYuOMG3bdpMgELZI96ThdXtHfSye7cU73z8Rhq5Tv+YPhHZWkf/9c8g7PbHX\nCYxGgkDY4uJDLxjTY2N15sxT+bB2U9zz/RPV/MJzdHy8kawjj6bw4kssrWUHCQIxadg17h9N8L13\naXn1FXyzDqHk+hsn1TThcCQIREIb7zp/q3Vv20r92kfxTCugfOly3D6fbbWtJEEgEpIT4/7RhGsC\n1Nx3D+7UVMqXr8STm2d7D1aRIBAJY998/6effM43wQBg77h/JNFgMDZNGA5Tvvx2fOUVjvViBQkC\n4aihxv2ZXvvH/SPRw2FqVt9NtKmJoquuJvOwRY72YwUJAmG70cb9p80/lmBLyOk2gdg0Yd2ahwjt\n+pr8c84j75TTnG7JEhIEwjZjHfenpniBxAiCpueepfOTTWQdfQyFFyXvPXwkCISlJnp+v5Pa3vk7\nra+/Strs2ZRclxzThMORIBCmS5T5/ono+moLDU88hqewkLLbViT0ZcbMIEEgTJFo8/0TEQ5UU3v/\natw+H+XLbseTk+N0S5aTIBATkojz/RMRbWsjcPed6JEIFSt+ga9s8t9qbSwkCMS4TeZx/0j0cJjA\n6ruJtjRTfPV1ZCxY6HRLtpEgEGOSDOP+kRi6Tu1DDxDevYtpP/ghuSd/1+mWbCVBIIa1b9z/+c4v\n2bh386Qe94+m6Zmn6fpsM9nHHU/Bj37sdDu2kyAQ35Js4/7R1L76Oq3r15E2Zy7F11yX1NOEw5Eg\nEMDI4/6zF34XbyjD4Q6t0fnF59Q89DBefxFlty3D7U3uacLhSBBMYWMd9/uzs2kM2Xcbb7uE935D\n7e/vx5ORETubMDv5pwmHI0EwxSTTfP9ERFpbCay6EyMaYf6//Su9xaVOt+QoCYIpYqqN+0eih0LU\n3HMX0dZWSq67gdxFh03ocubJQIIgiSXrfP9EGLpO7YP3E/5mD9N++CNyTvyO0y0lBAmCJJPs8/0T\n1fj0U3R98TnZJ5xIwQUXOt1Owog7CJRSvwQuALzAfZqmPWJaV2JcZNw/Nq1vrqftrfWkzzuU4p9d\nO6UD8WBxBYFS6lTgRE3TTlJKZQK/MLctMRYy7h+7zs820/j0k3iLiym7dRlurwTjgeLdIzgb2KKU\negHIBv7FvJbEaN6oepdXt78t4/4xCu3ZTe2D9+POzKR82e2kZGU53VLCiTcICoEZwPnAbOAlYP5o\nT/L7s+MsFx+76rndLlvr7ajcTbA3yJmzT+aUWcejCudYvps7WX924cYmdq2+G3Sdw379S3IWzrW0\n3ljZXW808QZBM7BN07QosEMpFVJKFWqa1jTSk+ycovH7s22rp+sGbrfLlnq769op6FnCb086f2Dc\n39TUaWlNO7+XZtbTQz1881+/I9LaSskNNxH2Vwy53cn6+sZTbzTxLqp+DzgHQClVBmQQCwdhIcMw\n+NOblYR7UuTg3yiMvj5qHrif3uq9FFx4ETnHn+B0SwktriDQNO2vwGal1EbgReAWTdMMUzsT3/KJ\n1siO6iCdPRGnW0lohmHQ8NQTdG/5gpyTvsO0H/zQ6ZYSXtzTh5qm/dLMRsTIItE+/vx2FYAEwSja\n1r9B8O9/I13Np/iqa2SacAym3vmWk9Sbm6ppCsYu8d3VLUEwnM7Nn9D4zJ/wlpRQdstSXB5ZMzcW\nEgSTQLCrl5ff3z3wcWdPr3PNJLDQ7l3UPvR7UrKyKF9+OymZso5irCQIJoHXP9pDWeH+N7UMDb4t\n0txE4J67QNcpu205qf4ip1uaVCQIJoGfnDaX80+cBcD5J82ioiix5qCd1tfdTWDVXfQFg5RcdyPp\nc4ZeKyCGJwOoScDtdlEZaAPgGOXniPkltLV2OdxVYjCiUWofuJfeQDWFF11M9rHHOd3SpCR7BJPE\nzuogvtQUKvxZeD3yY4P+acIn19K99StyTj6F/HN/4HRLk5a8oyaBaJ/OrroO5pTlDCxnFtC67jWC\n775DxoKFFF95lUwTToAEwSSwp66DSFRnbnmu060kjI5NH9P07J9JLSuj9OZbZZpwgiQIJoGqQBCA\neRV5DneSGHq+3kndww+Skp1D+bKVpGTINOFESRBMAlXVQVwumF02da+yu0+ksZGae+4GoGzpCryF\nctq1GSQIEpxhGFQGglT4s0j3Te3d377uLgKr7qSvs4OS628kffZsp1tKGhIECa4xGKK9q5e5FVP7\n+IARjVJz32p6a2sovPgSso8+1umWkooEQYKrqo6tH5jKBwoNw6D+8Ufp2b6N3FNPI//75zjdUtKR\nIEhwVdX9BwqncBC0vvZX2t/7BxmHLaLo8n+SaUILSBAkuKpAkLysVApy05xuxREdGz+i6blnSS2v\noPSmW3GlpDjdUlKSIEhg3aEIgcYu5pbnTsm/gj1VldSteYiU3NzYNGF6utMtJS0JggS2s6YdA5g7\nBdcP9DY0ULN6FbjdlC9dibegwOmWkpoEQQIbOD4wxWYMIh0dBFbdQV9XJ6U33ETarFlOt5T0pvbE\ndIKrCgRJ9biZXjR1rsNvRKNs/687idTV4b/0p2QdeZTTLU0JskeQoPp0na9r2jmkNAdPytT4MRmG\nQf2jj9C+5StyT/8eeWd+3+mWpgzZI0hQ1Q1dhCN9U2ohUcsrL9H+wQbyjz6Kwssun5IHSJ0yNf7U\nTEKVU2whUfuH79P84vP4pk/n0H++XaYJbSZBkKD2nXE4ZwoEQfcOjfo/riElL4+ypSvxZMg0od0k\nCBJUVSBIaUEGWenJfUej3vo6au5dBSkplC9biXfaNKdbmpIkCBJQS3uIlvZw0k8b9nV2Erj7TvTu\nbkpvvJm0GTOdbmnKkiBIQJX96wfmlifvQiI9EqHm3lVEGurx//QKshYvcbqlKU2CIAHtOz6QrDMG\nhmFQ/8eH6ancQd6ZZ5F/xplOtzTlSRAkoKrqINkZXorzk/OgWfNLL9Dx0YdkLl6C/5KfOt2OQIIg\n4YR6o+xt6EzaE43a399Ay8sv4psxk9IbbsLllrdgIpCfQoLZVdOObhhJuX6ge/s26h5dgyd/GuXL\nVuBOm5qnViciCYJx2FkTpLqx09IalUl6fKC3toaa++7B5fFSvmwlnrx8p1sSB5gSQfDk+h20d/WO\n+thoolGdp9+qRNeNEbcRz7b3qaoO4klxMaskee5vGO1oJ7DqTvSeHspuugXf9OlOtyQOMqFzDZRS\nRcAm4ExN03aY05L5enqjRKL6qI9B7K/+2tc1GGJ4Honq1Lf0sGVXM0fMKRx2G8M9PhrdMNhZE2Rm\nSTZeT3IssdUjvdSsXkWksZGiK64i8/AjnG5JDCHuIFBKeYAHgG7z2nHenLJc/uPab99IM9Qb5Y6n\nP+f2Sw9l4SxrVr/VNHbRE+5jXpKsHzB0nfo1fyC0s4r8s84m7/QznG5JDGMiQ4P/Ae4HakzqJaEZ\nBlz5fetCAJLv+EDzC8/R8fFGMo88isKfXOp0O2IEcQWBUupqoEHTtPUMuROdfNJ9HmYUWztu33dF\nomQ40Sj43ru0vPoKvlmHUHr9z2WaMMHFOzS4BtCVUmcBS4DHlFIXaJrWMNKT/H57D4Dtq5eW5mVa\nQSb+/IyBzw312Hjt28a+OxSPVG8sdtW1U1qYydxZo1+fz6nv5Vi0ff4FDWsfxecv5Ij/+DWp+eOf\nIUjk1zcZ640mriDQNO3Uff9WSr0N/Hy0EABobOyIp1xc/P7sgXqhUISW5i5c0b6Bzw/1WH1rN+s/\n3ktzMMSJi0o4bkHxwOfe+qSar3a1sOziI761DV03cLtdI9YbTbAzTF1zN99ZVDLq9+nA12aH8dQL\n1wTY+3/+L67UVEpuW0Ew6oFx9prIr2+y1huNGftrhgnbcJxhGLz+0TdcfuahLDxkGn/9YM+gz3/w\nVd3AX34rDFx/YBIfH4gGg7FpwnCY0ptuxVde4XRLYowmfKkyTdOS4lDw1j2tLDqkALfbxZavWyie\ntn+3PhzpY09dByd+r8Sy+pWT/I5GejhMzeq7iTY1UXTV1WQetsjplsQ4yBGcfhWFmRw5r5DWjjBb\nd7dw4mH7hwWV1W3ohoGabt20XlUgSIbPQ2lhpmU1rGLoOnVrHiK062vyzzmPvFNOc7olMU4SBP1y\ns3y43S4+3laPz5vC4bP3H7DbsTdIZpqXCosuK97bv8cxpzwX9yQ80ajpuWfp/GQTWUcfQ+FFFzvd\njoiDBMFBtuxuYf7M/EGXEN/xTStqRmxvoLGtx/Sau+s66NONSbl+oO2dv9P6+qukzZ5NyXU3yjTh\nJCU/tYO0tIcpOeD4QCTax666DubPiE2BvbFxr+k1912xeLIdH+j6agsNTzyGp7CQsluX405Ndbol\nEScJgoMU56fTFYoMfPzy+3vQdQN/XjqBxk5KCuJfdzCcnYF23C4Xh5TmmL5tq4QD1dTevxq3zxc7\nmzB3coWYGExucHKQy743j7XrNJ5cvwNccMriMtJ9Kfzt02oKctO47Ix5ptYzDIOqQJAZxVn4UifH\niUbRtrbYRUcjESpW/AJfWbnTLYkJkiA4iD8vndsvHXwhzQp/Fuceb80VdutauunsiXDCAbMUiUwP\nhwmsvptoSzPFV19LxoKFTrckTCBDA4dVDVyxOPF3rQ1dp/ahBwjv3sW0884n9+RTnG5JmESCwGH7\nzjicV5H4px43PfM0XZ9tJvvY4yi48CKn2xEmkiBwWFV1kIKcNPKzfU63MqK2t9+idf060ubMpfja\n62WaMMlMiZ9mus+D1+Me9TEztjuebXd091LX0p3w6wdaNn1Cw5OP4/X7KbttGW6vTBMmmylxsPDy\nMw8d02NmbHc8294ZaAcS+/hAeO837P3vO3CnZ1C+/HY82ZNnilOM3ZTYI0hUlYH+hUQJukcQaW0l\nsOpOjGiUsltuI7Wk1OmWhEUkCBy0szqILzWFcn/inWikh0LU3HMX0dZW5t56MxnzFzjdkrCQBIFD\non06u+o6mFOWQ0qCHXgzdJ3aB+8n/M0epv3wRxSdcZrTLQmLJdY7cArZU9dBJKon5PGBxqefouuL\nz8k+/gQKLrjQ6XaEDSQIHJKodzxufXM9bW+tJ33eoRRffV1S3n9RfJsEgUOqqoO4XLH7KCSKzs82\n0/j0k3iLiym7dRlur9fploRNJAgcYBgGlYEgFf4s0n2JMYMb2rOb2gfvx52RQfmylaRkWXMRFpGY\nJAgc0BgM0d7VmzDHByItzQRW3QW6Ttmty0gttu7ajCIxSRA4oKr/QiSJcHxAD/UQWHUXfcE2iq+5\njoxDlS11e77eSThQbUutRKyfaCQIHFCVIFcsNvr6qHngfnqr91Lwox+Tc/yJoz6n4akniLa3j/rY\nqLUjERqffgpDH//NYs1wcP3hXkM8r20ySowB6hRTFQiSm5VKQW6aYz0YhkHDU0/QveULck78DtPO\nv2BMz9N7ejAikVEfg9hf3Ya1j8IQMw9Gby+9DfV0bfmSrCMWx/ciRjHW+hSfPOxrGO7xZCNBYLPu\nUIRAYxdHK7+jU3Nt698g+Pe/ka7mU/yzayzpJX32HGb++//+1uN6KET1nf9Dxcp/HteFTYy+Plwp\nY7+Kk9n1k5kMDWy2s6YdA5jr4PUHOjd/QuMzf8JbUkLZzbfh8tj798AwDIqvvGrcv4QNTzxGtK3N\nsfrJTPYIbOb0FYlCu3dR+9DvScnMonzZ7Y5ME6akp5Myfca4n2dE+zD6xn4/SbPrJzPZI7BZVSBI\nqsfNjGL7fwEjzU0E7umfJrxtGalFRbb3IBKTBIGN+nSdr2vaOaQ0Z9ANVGyp3d3dP00YpOTaG0if\na+7VmA/WW19P/RNrCay6k46NHw36XOvf3iSw+m5L6w/0UVc3ZB/BDf9g5y+Wo4fMv2HNZCRBYKO9\nDZ2EI322rx8wolFqH7iX3kA1hRddTPZxx1tbzzBoXfcaRT+9goyFi2h57ZVBn+/48H1cLuvfeoau\n0/rG60P3YUBfezvhmlrL+5gMJAhs5MTxAcMwaHhyLd1bvyLn5O+Sf+4PLK/ZvW0rGYsOx+V207Xl\nS7wHrFTUw2FCe/aQviDe6xsYY+9j6xYyj1g8ZB+5J3+XtNlzSMmWpdQgBwttte+Mwzk2BkHrutcI\nvvsOGQsWUnzlz2yZsvSVl5OSnUO0rZXubV9RdvNtA5/rqaoEXR9xBWPl3fcQrPx68Py/YRBpaSa0\ne9fgWQ7DAJeL4quvJW3G4HtP+CpmkJKTQ6S1le6tWyi7bfngz8+YSapfjpOABIGtKquDlBZkkJVu\nz1l9HZs+punZP5NaWkbpzbfaNk3oyY1NjXZs3Ijb5yNz0eEDn+up1EjJzMJXMX3Y589bvpTGxo5v\nPV73yMMUXHAh3oKCIZ41RB95/X18sAF3evqgPgBSMs2/fd1kJUMDm7S0h2jtCNt2fcKer3dS9/CD\npGTnxM4mzLD/cmhdW7eQoRYMCqAeTSNdxfYGIo2N49ugMfZhwYG6t20lQ80fdAn27m1bSZs9J67t\nJaO4gkAp5VFKPaaUelcp9aFS6odmN5ZsKgeOD1i/kCjS2EjNPbGj8mVLl+P1+y2vOZRoSzPekgOO\nD0R6Ce3eRbqaD0Dr+tft6aOtDW/R4FvKdX7xOZmHW7O0eTKKd4/gSqBJ07RTgPOA1ea1lJwGDhRa\nvEfQ191FYNWd9HW0U3L9jaQ7+FfPW1SM3tU18HHLX1/G0HVS/UWEAwG8Nl0V2VdRQTS4f0Vi8L13\nyVpypFx96QDxDhr/DDzT/28XkPxnZUxQVSBIdoaX4vx0y2oY0Sg1962mt7aGwosvIfvoYy2rNRZF\nl15O/eOP0vDUE+CC3O+eijstnba/vYmnsBD/JT+1pQ//pZfT8ORa6p94DLfHS/qhioz+vRIRE1cQ\naJrWDaCUyiYWCL82s6lkE+qNsrehk8VzCyz7K2QYBvWPP0rP9m3knnIa+Wefa0md8fD6/VSs/OdB\nj/nKK+Cc8+LbYJzfO09eHmW3LI2v5hQR92FkpdR04DlgtaZpT4/lOX5/drzl4mJXPbfbNWK9z3c0\nohsGiw8tMq2ng7dT/exztL/3D/KWLGbB8ptxmzxDsK9eW5qXgoJMfAfUH+oxs+odKP3iH5E5c/q4\nzkAcq7RhXoMVrw3s/10YTVzvFqVUMbAOuFXTtLfH+ryhpoSs4vdn21ZP1w3cbtew9TZtja1eK81P\nM6Wng19bx8aPqF37BKnlFRRc+3OaW81dNntgvVAoQnNzF172X0thqMfMqjdIdiE9Ld2m1Di43nCv\nwezXtq+e3b8Lo4n3z8avgDzgN0qpfyO23OtcTdPCcW4vqVVVB/GkuJhVYv5fgZ6qSurWPERKbm7/\nNKHMjYvxi/cYwQpghcm9JCVdN9hZE2RmSTZej7m7tL0NDdSsXgVuN+VLV4x5oc1EuDPScR10mfOh\nHptshnsNyfDaxkJWFlqspqmLnnAf80xeP9DX2Ulg1R30dXVSdsttpM06xNTtD6fosivG9NhkM9xr\nSIbXNhaystBilRbc0UiPRKi57x4idXX4f3IZWUcebdq2xdQkQWCxfZcuN+tEI8MwqLr3AXp2aOSe\nfgZ5Z33flO2KqU2GBharCgQpyk8nNzPVlO21vPISzW//nYxFR1B02RWyOk6YQvYILBTsDNPYFjLt\n/gXtH75P84vPkzFrJmU33WzJfLqYmiQILLTvRKM5Jhwf6N6hUf/HNaTk5bHwN7/GnWbdUmUx9UgQ\nWGjfhUgmukfQW19Hzb2rICWF8qUr8BVaP00ophYJAgtVBYJk+DyUFsZ/LYC+zk4Cd9+J3t1N6Y03\nkzZzlnkNCtFPgsAivZE+9tR1MKc8F3ecB/T0SC81964i0lCP/7LLyVq8xOQuhYiRILDI7roO+nQj\n7vUDhmFQ/8gaeip3kPe9s8j/3lkmdyjEfhIEFqncd+vzOI8PNL/4PB0bPyRz8RL8l9pz3r6YuiQI\nLFJVHcTtcjG7NGfczw1ueI+WV17CN2MmpTfcNOhae0JYQd5hFjAMg6pAkBnFWfhSxzfX3719G/WP\nPYInfxrly1bgTnPu1uli6pAgsEBdSzddoei4hwW9tTXU3HcPLo+X8mUr8OTlW9ShEINJEFigMo4L\nlUY72mPThD09lN10Cz65W6+wkQSBBfYtJBrrHoHe20vN6lVEmhopuvyfyDz8CCvbE+JbJAgsUFUd\npCDHx7Sc0cf3hq5Tt+YPhHZWkX/W2eSdfoYNHQoxmASByTq6e6lr6WZuxdguRNL8wnN0btpI5pFH\nUfiTSy3uToihSRCYbGegHRjbsCD4j3doefUVfLMOofT6n8s0oXCMvPNMVhkY20Kirq1fUf/4Y3im\nFVC+dDlun8+O9oQYkgSByaqqg/hSU6goGv5Eo3AgQO39q3GnplK+fOXA3YOFcIoEgYmifTq7ajuY\nU5ZDyjC7+dFgkMCqO9DDYUpvujV25x8hHCZBYKI9dR1E+/RhhwV6OEzN6ruJNjdTdOVVZB62yOYO\nhRiaBIGJRlpIZOg6dQ8/SGjX1+Sfcx55p5xmc3dCDE+CwEQ7A0FcwJyybwdB01+eofPTT8g6+hgK\nL7rY/uaEGIEEgYkqA0HK/Vmk+wZfHLrtnbdpXfcaabNnU3LdjTJNKBKOvCNN0qcbtHf1Mu+gYUHX\nli9peGItnsJCym5djjvVnMuaC2EmCQKTRKI6MPj4QLh6L7UP3Ivb56N82Uo8uebd7UgIM0kQmCQS\n7QP2LySKtrURWHUneiRC2S1L8ZWVO9meECOSIDBJJKqTm5VKYW4aejhM4J67iLa0UPxPPyNjwUKn\n2xNiRBIEJtANg2ifHrt/gWFQ+9ADhPfsZtp555N78ilOtyfEqCQITBDdd3ygPJfGZ56m67PNZB97\nHAUXXuRwZ0KMTdw3QVVKuYD7gMVACLhe07SvzWpsMon0xYJgduAL2tavI23OXIqvvV6mCcWkMZF3\n6oWAT9O0k4BfAXeY09LkE+0z8Bp96K/+Ba/fT9lty3B7ZZpQTB4TCYKTgdcBNE37CDjGlI4mmT5d\nx4hESO8ZNM0QAAAFCElEQVQL4U7PoHz57Xiyx38JcyGcFPfQAMgBggd8HFVKuTVN04f64rvuugtd\nNyZQbnzcbpct9aJRnRR3lLDby4Yli/hg/V8tr2nXa5N6yVHv9ttXjvo1EwmCdiD7gI+HDYGBL3DH\ndw/AeNlRLzU1hVBPrI7HZ99wIBm/l1LPORMJgg3A+cCzSqkTgC9H+uIVK1bQ2NgxgXLj4/dn21Zv\n7do/4Ha7uOKK62ypZ+drk3qTv95YTCQIngfOUkpt6P/4GhP6EUI4IO4g0DTNAG42sRchhENkolsI\nIUEghJAgEEIgQSCEQIJACIEEgRACCQIhBBIEQggkCIQQSBAIIZAgEEIgQSCEQIJACIEEgRACCQIh\nBBIEQggkCIQQSBAIIZAgEEIgQSCEQIJACIEEgRACCQIhBBIEQggkCIQQSBAIIZAgEEIgQSCEQIJA\nCIEEgRACCQIhBBIEQggkCIQQgCeeJymlcoDHgRzAC/xC07QPzWxMCGGfePcIbgfe1DTtNOAa4F7T\nOhJC2C6uPQLgDiDc/28v0GNOO0IIJ4waBEqpa4GVgAG4+v9/jaZpnyilSoC1wDJLuxRCWMplGEZc\nT1RKHQ48Sez4wBumdiWEsFVcQaCUWgj8BbhE07QvTe9KCGGreIPgBeAIYDex4UKbpmk/Nrc1IYRd\n4h4aCCGShywoEkJIEAghJAiEEMS/oGhcnFqSrJT6MXCxpmlXWLR9F3AfsBgIAddrmva1FbUOqns8\n8F+app1ucR0PsAaYBaQCv9U07WUL67mBhwAF6MBNmqZttapef80iYBNwpqZpO6ys1V/vU6Ct/8Nd\nmqZdZ2GtXwIXEPudu0/TtEeG+1q79ghsX5KslLoL+C2xWQ2rXAj4NE07CfgVsRWXllJK/QuxXxaf\n1bWAK4EmTdNOAc4DVltc74eAoWnaycBvgN9ZWaw/6B4Auq2sc0A9H7HXd0b/f1aGwKnAif3vzdOA\n6SN9vV1BcAfw+/5/27UkeQNws8U1TgZeB9A07SPgGIvrAVQBdk3V/pnYLyTEAjViZTFN014Ebuz/\ncBbQamU94H+A+4Eai+vssxjIVEqtU0q92b9nZ5WzgS39U/0vAa+M9MWmDw3sXpI8Qr1n+lPRSjlA\n8ICPo0opt6ZpulUFNU17Xik106rtH1SrG0AplQ08A/zahpq6UuqPxPa2LraqjlLqaqBB07T1Sql/\ntarOQbqB/9Y07WGl1DzgNaXUoRa9XwqBGcD5wGxiYTB/uC82PQg0TVtDbFw5yEFLkt+zup5N2oHs\nAz62NAScoJSaDjwHrNY07Wk7amqadnX/2H2jUmqBpmlW7EFeA+hKqbOAJcBjSqkLNE1rsKDWPjuI\n7dGhaVqlUqoZKAUCFtRqBrZpmhYFdiilQkqpQk3Tmob6YluGBv1Lkv8MXJ5k5yVsIDZ2Ril1AmDn\ncmsrj30AoJQqBtYB/0vTtEdtqHdl/wEuiB187ev/z3Sapp2qadrp/QdcPwOusjgEAK4F/h+AUqqM\n2B+RWotqvQecc0CtDGLhMCRbZg2IHfTxAXf3H2lPliXJzwNnKaU29H98jY217VgS+isgD/iNUurf\n+mueq2laeOSnxe054BGl1DvE3pvLNU3rtajWgexaXvswsdf3D2KzItdatQepadpflVLfVUptJPZH\n4xZN04Z9nbLEWAghC4qEEBIEQggkCIQQSBAIIZAgEEIgQSCEQIJACIEEgRAC+P87wr0pY/DtmgAA\nAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x108f24050>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "u = [0,0,1,6]\n",
    "v = [0,0,4,2]\n",
    "u_bis = [1,6,v[2],v[3]]\n",
    "w = [0,0,5,8]\n",
    "plt.quiver([u[0], u_bis[0], w[0]],\n",
    "           [u[1], u_bis[1], w[1]],\n",
    "           [u[2], u_bis[2], w[2]],\n",
    "           [u[3], u_bis[3], w[3]],\n",
    "           angles='xy', scale_units='xy', scale=1, color=sns.color_palette())\n",
    "# plt.rc('text', usetex=True)\n",
    "plt.xlim(-2, 6)\n",
    "plt.ylim(-2, 9)\n",
    "plt.axvline(x=0, color='grey')\n",
    "plt.axhline(y=0, color='grey')\n",
    "\n",
    "plt.text(-1, 3.5, r'$||\\vec{u}||$', color=sns.color_palette()[0], size=20)\n",
    "plt.text(2.5, 7.5, r'$||\\vec{v}||$', color=sns.color_palette()[1], size=20)\n",
    "plt.text(2, 2, r'$||\\vec{u}+\\vec{v}||$', color=sns.color_palette()[2], size=20)\n",
    "\n",
    "plt.show()\n",
    "plt.close()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span class='pquote'>\n",
    "    Geometrically, this simply means that the shortest path between two points is a line\n",
    "</span>\n",
    "\n",
    "# P-norms: general rules\n",
    "\n",
    "Here is the recipe to get the $p$-norm of a vector:\n",
    "\n",
    "1. Calculate the absolute value of each element\n",
    "2. Take the power $p$ of these absolute values\n",
    "3. Sum all these powered absolute values\n",
    "4. Take the power $\\frac{1}{p}$ of this result\n",
    "\n",
    "This is more condensly expressed with the formula:\n",
    "\n",
    "$$\n",
    "\\norm{\\bs{x}}_p=(\\sum_i|\\bs{x}_i|^p)^{1/p}\n",
    "$$\n",
    "\n",
    "This will be clear with examples using these widely used $p$-norms."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The $L^0$ norm\n",
    "\n",
    "All positive values will get you a $1$ if you calculate its power $0$ except $0$ that will get you another $0$. Therefore this norm corresponds to the number of non-zero elements in the vector. It is not really a norm because if you multiply the vector by $\\alpha$, this number is the same (rule 4 above)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The $L^1$ norm\n",
    "\n",
    "$p=1$ so this norm is simply the sum of the absolute values:\n",
    "\n",
    "$$\n",
    "\\norm{\\bs{x}}_1=\\sum_{i} |\\bs{x}_i|\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The Euclidean norm ($L^2$ norm)\n",
    "\n",
    "The Euclidean norm is the $p$-norm with $p=2$. This may be the more used norm with the squared $L^2$ norm.\n",
    "\n",
    "$$\n",
    "\\norm{\\bs{x}}_2=(\\sum_i \\bs{x}_i^2)^{1/2}\\Leftrightarrow \\sqrt{\\sum_i \\bs{x}_i^2}\n",
    "$$\n",
    "\n",
    "Let's see an example of this norm:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 2.\n",
    "\n",
    "Graphically, the Euclidean norm corresponds to the length of the vector from the origin to the point obtained by linear combination (like applying Pythagorean theorem).\n",
    "\n",
    "$$\n",
    "\\bs{u}=\n",
    "\\begin{bmatrix}\n",
    "    3 \\\\\\\\\n",
    "    4\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\begin{align*}\n",
    "\\norm{\\bs{u}}_2 &=\\sqrt{|3|^2+|4|^2}\\\\\\\\\n",
    "&=\\sqrt{25}\\\\\\\\\n",
    "&=5\n",
    "\\end{align*}\n",
    "$$\n",
    "\n",
    "\n",
    "So the $L^2$ norm is $5$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The $L^2$ norm can be calculated with the `linalg.norm` function from numpy. We can check the result:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5.0"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.linalg.norm([3, 4])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is the graphical representation of the vectors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQIAAAECCAYAAAAVT9lQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFSBJREFUeJzt3XtwlfWdx/F3zkICCIRrsQaEFetPENRyGRWpiJR1g9HR\nWSsrIAOESluLghRGlrLMwghKQdiR7WW0kGK5WK1UOzB1EJK40pFLERYD/AqKKCbWpJUEgYSQ8+wf\nuRgwl5NznpznOef5vGacyQnn5HzmYD48l+/z/FIcx0FEgi3kdQAR8Z6KQERUBCKiIhARVAQigopA\nRIA2sbzYGLMfOF3z8IS1Njv2SCISb1EXgTEmDXCstXe5mEdEPBDLFsFNwBXGmDeBfwIWWGt3uxNL\nROIplmME54CfWWvvBn4IbDDG6JiDSAKK5Rf3r8AGAGvtMeDvwDfdCCUi8RXLrsE0YDDwmDHmKqAT\nUNTYkx3HcVJSUmJ4u+S3evVqAGbNmuVxEkkyzf7ixVIEvwbWGWP+FwgD06y14UaTpKRQXHwmhrdz\nX8+enXyVKRx2CIX0OUXKj7n8mqk5UReBtbYSmBTt60XEP3RwT0RUBCKiIhARVAQigopARFARiAgq\nAhFBRSAiqAhEBBWBiKAiEBFUBCKCikBEUBGICCoCEUFFICKoCEQEFYGIoCIQEVQEIoKKQERQEYgI\nKgIRQUUgIsS20hHGmG8A+4DvWmv/6k4kEYm3qLcIjDFtgF9SvSqyiCSwWHYNVgC/AApdyiIiHomq\nCIwxU4DPrbXbiWClVRHxt2iPEUwFwsaYscDNwHpjzH3W2s+belEkq7LGm58yhULVneqnTLX8mAn8\nl8txHN9likRURWCtHVX7tTEmF5jRXAkAvlwu2k+ZtCx6y/gp12efFbFu3QusWPEsJSVfeh3nEpEU\nkxunDx0XfoZIQqqqqmLt2he4/fbhZGT0IRRq+FeqsrKSI0eOxDld5GIuAmvtXTp1KEFUUPA+WVlj\neeqpOaSlpfHQQw83+tydO7ezadP6OKZrGQ0UibTQuXPnWLJkEWPH3sFf/rIPgOnTZ9CuXbsGn3/x\n4kXKyyvo27cfx475899MFYFIC5w9e5Yf/ej7PP/8Ki5evAhAhw4dmDIlu9HX5Oa+xejRd9G3bz8K\nCg7FK2qLqAhEWuCKK67g+ed/wfXXD6j73sMPT6Jbt+6NviYcdrjiio4A9O//LU6d+qTVc7aUikCk\nBSorK5k27RGOHj3CtGnfp23btsyY8ViTr7n77sy6rwcPvpHevfu0dswWi+laA5EgcRyHn/zkCfLz\nc5k4cTLLlq1g6NDh9Ov3z15Hi5m2CEQitHLls2za9FtGjx7D8uWrSElJ4Xvf+3evY7lCRSASgc2b\nN7B8+VIGDbqRX/96PW3btvU6kqtUBCLNyM/P5cknZ5KR0ZuNG1+hY8fEGyFujopApAkFBe8zdeok\n2rfvwMaNr3Llld/0OlKr0MFCkUYUFn7KhAkPUlFRzubNrzFgwECvI7UaFYFIA86cKWPChO9RVFTI\nmjW/4jvfGdX8ixKYdg1ELlM7K3D48Ps89dRPm7yGIFmoCETquXxWYPbsuV5HigsVgUg9Dc0KBIGK\nQKRGss8KNEVFIEIwZgWaoiKQwAvKrEBTdPpQAi1IswJNURFIYAVtVqAp2jWQQArirEBTVAQSOEGd\nFWiKikACJ6izAk1REUigBHlWoCkqAgmMoM8KNCXqswbGmBDwAmCAMPADa+1ht4KJuEmzAk2LZYvg\nXsCx1o4EFgJL3Ykk4q76swI5ORsCOyvQlKiLwFr7OvBozcN+wBduBBJxU/1ZgVWr1gR6VqApMQ0U\nWWvDxpgc4H7gQVcSibhEswKRi3my0Fo7xRjzDWCPMWaAtfZ8Y8/147rxfsoUClWfxvJTplp+zASN\n53Ich+zsbPLzc8nOzmbp0sVxO03YUKaqqirS0zv49nOM5WDhJKC3tfYZoByoqvmvUX5Zy75Wz56d\nfJUpHHYIhVJ8lQki+5wqKspJTU2L6zn5pnKtWPEM69atY/ToMSxevJySki89zVRVVUVp6TlP/m4j\nKZ9YtgheA9YZY/Jrfs4T1toLMfw8SWC5uTu4eLGSrKz7vY6iWYEoRF0E1tpzwHgXs0gCePnlDezZ\ns5vL/+E/c+YMH3xwjNtuG0n37j28CYdmBaKlqw+lRcaPn8j48RMv+V55eTkzZ85g2bKVnpaAZgWi\npyKQmKWkpLBw4X9x9dX9PMug+wrERkUgMUtLS/O0BHRfgdjpWgOJWH7+TnJyXmTOnMcpKyur+/6O\nHduZN2+2J5k0K+AOFYFEpKSkmKKiQqZMmc5nnxVy4MD+uj/Lzd1Ohw4d4p5J9xVwj4pAIrJ3727G\njbuX48ePcerUJwwceEPdnx08eIBhw4bHPdOSJUt0XwGXqAgkIpmZWXTunM62bW8wdOhwevToCcCJ\nEx9SWnqaIUPiWwSbN29g0aJFmhVwiYpAWiQvbyd33jmm7vHBg/vp1etKrroqI24ZamcF+vTpo1kB\nl6gIJGJlZWUUF3/OgAFf7RYcOPAeQ4YMA2D9+rWtnqH+rMC2bds0K+ASFYFErF27drRtm1q3L/7x\nxyfZu/dd+ve/lqKiQrp06dqq73/5fQUGDRrUqu8XJJojkIilpqYyd+58cnJe5LrrDF27dmPevAVs\n3foGxcXFzJjxWKu9t2YFWpeKQFokMzOLzMysS743atRdrfqemhVofdo1EF8L+qxARUU5juO0+vuo\nCMTXgr4GQW7uDrZufb3V30e7BuJbQbqvgNeXd6sIxJeCdl8Bry/vVhGI7xw+XMC0aY8E/r4C8by8\nW0UgvlJUVMiECQ9SXn4+8PcViOfl3TpYKL5ROytQWPhpoNYg8MPl3SoC8YXKykqysydTUHAoULMC\nfrm8W0UgnnMch7lzZ5GXtzNwswJ+ubxbRSCee+655Wzc+FIgZwX8cnm3ikA8tXnzBp599ulAzAo0\nxevLu1UE4pm3384L1KxAYyK9vLuiopwtW15l0aL5XLx4EYBlyxbz8ccfxZwhqiIwxrQxxqw3xrxt\njHnXGHNvzEkkUA4fLtAaBDUivbx71653uOee+zh+/FhdEezdu5tevWL/7KKdI5gElFhrJxtjugHv\nAX+MOY0EgmYFLhXp5d233XY7J0+eICOjD+3atePTT0/RvXsP0tLSYs4QbRH8Dnil5usUoDLmJBII\n9WcFdF+Br0RyeXf79u3585/f4fbbvwNAQcEhBg260ZX3j6oIatY9xBjTiepCWOBKGklqQZ0VcNM/\n/vH3umMJ+/bt4ZZbRrjyc2NZFr0P1Ssir7HWvhzJa/y4NryfMoVC1fuIfspUK9ZMjuMwffp08vJ2\nkp2dzdKli105TZgon1VVVRXp6R1izvvQQ//G1q1bqaw8S37+TubNm+PKZxBVERhjegFvAo9Za3Mj\nfZ0Xa8M3pbG17L0SDjuEQim+ygTufE4rVz7L2rVrGT16DIsXL6ek5Etf5HJbY5mqqqooLT0Xc96M\njP48+ujjFBUV0rVrd9q06djsz4ykKKI9fTgf6AIsNMbkGmN2GmNiP2IhSUmzAu44dOggTzzxIwA2\nbXqJqVOnu/azoz1GMAuY5VoKSVqaFXBPt27dGTZsOH/4w++55pr+3H33ONd+ti5DllajWQF3ZWT0\n5pFHprbKz1YRSKvQrEBiURGI6zQrkHh0rYG4SrMCiUlFIK4J8n0FEp2KQFwT5PsKJDoVgbhCswKJ\nTUUgMdOsQOJTEUhMNCuQHHT6UKKmWYHkoSKQqGhWILlo10BaTLMCyUdFIC2iWYHkpCKQFtGsQHJS\nEUjENCuQvFQEEhHNCiQ3FYE069ChQ5oVSHI6fShNKioq5J57xmlWIMmpCKRRtbMCp06d0qxAktOu\ngTSo/qzAkiVLNCuQ5FQE8jWXzwosWKD1a5KdikC+RrMCwaMikEtoViCYVARSR7MCwRVTERhjbjHG\nRLzkmfiX7isQbLEsgjoXeASIfRE78ZTuKyCxbBEcBx5wK4h4o/59BVatWqNZgYCKugistVuAiy5m\nkTjTfQWkVlwnCxNlLXuvhELVp+nikclxHKZPn05e3k6ys7NZunRxk6cJ/fQ51efHXA1lqqqqIj29\ngy/zgjtFEPFJ5kRZy94r4bBDKJQSl0wrVz7L2rVrGT16DIsXL6ekpPFDPX77nGr5MVdjmaqqqigt\nPedJ3kjKx43Th44LP0PiSLMCcrmYtgistSeBES5lkTjQrIA0RANFAaJZAWmMLkMOCM0KSFNUBAGg\nNQikOdo1SHKaFZBIqAiSmNYgkEipCJKY7isgkVIRJCnNCkhLqAiSkGYFpKVUBElGswISDZ0+TCKa\nFZBoqQiShGYFJBbaNUgCmhWQWKkIEpxmBcQNKoIEp1kBcYOKIIFpVkDcoiJIUJoVEDepCBKQZgXE\nbTp9mGA0KyCtQUWQQDQrIK1FuwYJQrMC0ppUBAlAswLS2lQECUCzAont5MmP6r4+f/48f/vb37wL\n0wgVgc9pViDxHTiwv+7rHTu207VrVw/TNExF4GOaFUgOgwffxMGDB7hwoYI2bdqQmprqdaSviWVZ\n9BTg58BNQDkw3Vr7oVvBgk6zAsnjmmv6s2XLqxw9epg5c57yOk6DYtkiuB9Is9aOAOYDz7kTSerP\nCuTkbNCsQBIYPPgmOnXq7MutAYitCEYCfwKw1u4GhrmSKOAcx6mbFVi1ao1mBZLEtdd+ix//eJbX\nMRqV4jjRrWFqjHkBeNVa+2bN44+Aa6y14Yaev3r1aicc9td6qaFQCn7K9OWXZwiHw5SWlpKenk7n\nzuleRwL89znV8mMuP2Z68snZzZ5mimWysAyof/Qq1FgJ1D0h5L/TXn7M1LFjR7p06eJ1jEv48XMC\nf+byY6bmxFIEu4As4FVjzK3AoaaePGvWrIRZy94rL730IqFQChMmTPPVrIDfPqdafszlx0yRiKUI\ntgBjjTG7ah5PdSGPgK9KQIIh6iKw1jrAD13MIiIe0UCRzzmOwzvvvM3Ro0e8jiI13n33zxw69H9e\nx3CVisCnagvgjTe2YMwArr9+gNeRpMatt46gffv2vP76a0lTCLofgU+tXv0zLlyo5Oabh7B//z5P\ns6Snt6e09LynGRrida7U1DR+85u13HHHndx33/2e5XCDisCnZs+ex8mTH3Hw4Hv06XM13/72UM+y\n+PVIuFe5wuEwb7+dx5kzZfz0p4vo0sV/FxG1lIrAx/r27Uffvv04efIjjh49ot0Dn9i7dw833/zt\npCiAWiqCBNC3bz+vI0g9t9xyq9cRXKeDhSKiIhARFYGIoCIQEVQEIoKKQERQEYgIKgLxoYqKcqK9\nc5ZER0UgvpObu4OtW1/3OkagaLJQPPPyyxvYs2c3l9+H5cyZM3zwwTFuu20k3bv38CZcwKgIxDPj\nx09k/PiJl3yvvLycmTNnsGzZSpVAHKkIxFdSUlJYuPC/uPrqfl5HCRQdIxBfSUtLUwl4QEUgnsjP\n30lOzovMmfM4ZWVldd/fsWM78+bN9jBZMKkIJO5KSoopKipkypTpfPZZ4SWrBefmbqdDhw4epgsm\nFYHE3d69uxk37l6OHz/GqVOfMHDgDXV/dvDgAYYNG+5humBSEUjcZWZm0blzOtu2vcHQocPp0aMn\nACdOfEhp6WmGDFERxJuKQDyTl7eTO+8cU/f44MH99Op1JVddleFhqmCK6fShMeYB4EFr7cRmnyxS\nT1lZGcXFnzNgwFe7BQcOvMeQIdWLaq9fv5bJk6cBsG/fHr744h8UFLzP4ME3cuLEh2Rk9Gby5Ic9\nyZ6Mot4iMMasBp4GtD6XtFi7du1o2za1bnm3jz8+yd6979K//7UUFRXW3Rj09OnTXLhwgbFj/5Wz\nZ78kHA7Tt28/0tP9tUhsoot1EdQtwAyXskiApKamMnfufHJyXuS66wxdu3Zj3rwFbN36BsXFxcyY\n8RgA7du3Z8SIkQBYe4SZM2f7Zrn4ZNJsERhjpgGzAYfqf/0dYKq19hVjzKhWzidJLDMzi8zMrEu+\nN2rUXZc8TktLA6q3DBzHoXPndBzHoaKiAugUr6hJr9kisNauBda68WY9e/rvL85PmUKh6s1kP2Wq\n5VWmt956i7Nnz+I4DjfcMJCePTuRl5fHoEGDPM3VFD9mak5crzXw22o5flvBJxx2CIVSfJUJvP2c\nPvmkiMLCQvr3v5bU1PasX7+J7t174DjVWwr6rJoXSTHpoiPxtaysr9YUHDPmXzxMktxiKgJrbT6Q\n71IWEfGIBopEREUgIioCEUFFICKoCEQEFYGIoCIQEVQEIoKKQERQEYgIKgIRQUUgIqgIRAQVgYig\nIhARVAQigopARFARiAgqAhFBRSAiqAhEBBWBiKAiEBGiXNfAGNMZ+C3QGWgLzLHWvutmMBGJn2i3\nCJ4E3rLW3glMBf7HtUQiEnfRrnT0HFBR83Vb4Lw7cUTEC7Esi/4XY8yVwEvA462aUkRaVdTLohtj\nBgMbqT4+8E4rZBOROElxHKfFLzLGDAR+DzxkrT3keioRiatoi+APwI3AR1TvLpy21j7gbjQRiZeo\nikBEkosGikRERSAiKgIRQUUgIkQ/Wdgifr82wRjzAPCgtXaihxlSgJ8DNwHlwHRr7Yde5anPGHML\n8Iy1drQPsrSheq6lH5AKPG2t/aPHmULAC4ABwsAPrLWHvcxUyxjzDWAf8F1r7V8be168tgh8e22C\nMWY18DTVp0G9dD+QZq0dAcyneozbc8aYuVT/T57mdZYak4ASa+0dwDhgjcd5AO4FHGvtSGAhsNTj\nPEBdaf4SONfcc+NVBM8Bv6r52m/XJuwCfuh1CGAk8CcAa+1uYJi3ceocB/w0I/I7qn/ZoLq8Kz3M\nAoC19nXg0ZqH/YAvvEtziRXAL4DC5p7o+q6BX69NaCLXK8aYUfHO04DOQGm9xxeNMSFrbdirQADW\n2i3GmL5eZqjPWnsOwBjTCXgFWOBtomrW2rAxJofqLbsHPY6DMWYK8Lm1drsx5j+ae77rReDXaxMa\ny+UjZUCneo89LwG/Msb0AV4D1lhrX/Y6Ty1r7ZSaffI9xpgB1lovt3ynAmFjzFjgZmC9MeY+a+3n\nDT05XgcLB1K9SadrExq3C8gCXjXG3Ar47XPy+hgKAMaYXsCbwGPW2lyv8wAYYyYBva21z1B9oLeq\n5j/PWGvrtnKNMbnAjMZKAOJUBFQfPEkD/rvm6LiuTfi6LcBYY8yumsdTvQzTAL/Mos8HugALjTH/\nSXWuTGttRdMva1WvAeuMMflU/049Ya294GGeyzX7d6drDUREA0UioiIQEVQEIoKKQERQEYgIKgIR\nQUUgIqgIRAT4f1qiCtsaqcinAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x108f24350>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "u = [0,0,3,4]\n",
    "\n",
    "plt.quiver([u[0]],\n",
    "           [u[1]],\n",
    "           [u[2]],\n",
    "           [u[3]],\n",
    "           angles='xy', scale_units='xy', scale=1)\n",
    "\n",
    "plt.xlim(-2, 4)\n",
    "plt.ylim(-2, 5)\n",
    "plt.axvline(x=0, color='grey')\n",
    "plt.axhline(y=0, color='grey')\n",
    "\n",
    "plt.annotate('', xy = (3.2, 0), xytext = (3.2, 4),\n",
    "             arrowprops=dict(edgecolor='black', arrowstyle = '<->'))\n",
    "plt.annotate('', xy = (0, -0.2), xytext = (3, -0.2),\n",
    "             arrowprops=dict(edgecolor='black', arrowstyle = '<->'))\n",
    "\n",
    "plt.text(1, 2.5, r'$\\vec{u}$', size=18)\n",
    "plt.text(3.3, 2, r'$\\vec{u}_y$', size=18)\n",
    "plt.text(1.5, -1, r'$\\vec{u}_x$', size=18)\n",
    "\n",
    "plt.show()\n",
    "plt.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, the vector is in a 2-dimensional space but this stands also for more dimensions.\n",
    "\n",
    "$$\n",
    "u=\n",
    "\\begin{bmatrix}\n",
    "    u_1\\\\\\\\\n",
    "    u_2\\\\\\\\\n",
    "    \\cdots \\\\\\\\\n",
    "    u_n\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "$$\n",
    "||u||_2 = \\sqrt{u_1^2+u_2^2+\\cdots+u_n^2}\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The squared Euclidean norm (squared $L^2$ norm)\n",
    "\n",
    "$$\n",
    "\\sum_i|\\bs{x}_i|^2\n",
    "$$\n",
    "\n",
    "\n",
    "The squared $L^2$ norm is convenient because it removes the square root and we end up with the simple sum of every squared values of the vector. \n",
    "\n",
    "The squared Euclidean norm is widely used in machine learning partly because it can be calculated with the vector operation $\\bs{x}^\\text{T}\\bs{x}$. There can be performance gain due to the optimization See [here](https://softwareengineering.stackexchange.com/questions/312445/why-does-expressing-calculations-as-matrix-multiplications-make-them-faster) and [here](https://www.quora.com/What-makes-vector-operations-faster-than-for-loops) for more details.\n",
    "\n",
    "### Example 3.\n",
    "\n",
    "$$\n",
    "\\bs{x}=\n",
    "\\begin{bmatrix}\n",
    "    2 \\\\\\\\\n",
    "    5 \\\\\\\\\n",
    "    3 \\\\\\\\\n",
    "    3\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\bs{x}^\\text{T}=\n",
    "\\begin{bmatrix}\n",
    "    2 & 5 & 3 & 3\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\begin{align*}\n",
    "\\bs{x}^\\text{T}\\bs{x}&=\n",
    "\\begin{bmatrix}\n",
    "    2 & 5 & 3 & 3\n",
    "\\end{bmatrix} \\times\n",
    "\\begin{bmatrix}\n",
    "    2 \\\\\\\\\n",
    "    5 \\\\\\\\\n",
    "    3 \\\\\\\\\n",
    "    3\n",
    "\\end{bmatrix}\\\\\\\\\n",
    "&= 2\\times 2 + 5\\times 5 + 3\\times 3 + 3\\times 3= 47\n",
    "\\end{align*}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[2],\n",
       "       [5],\n",
       "       [3],\n",
       "       [3]])"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = np.array([[2], [5], [3], [3]])\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[47]])"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "euclideanNorm = x.T.dot(x)\n",
    "euclideanNorm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "47.0"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.linalg.norm(x)**2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It works!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Derivative of the squared $L^2$ norm\n",
    "\n",
    "Another advantage of the squared $L^2$ norm is that its partial derivative is easily computed:\n",
    "\n",
    "$$\n",
    "u=\n",
    "\\begin{bmatrix}\n",
    "    u_1\\\\\\\\\n",
    "    u_2\\\\\\\\\n",
    "    \\cdots \\\\\\\\\n",
    "    u_n\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\norm{u}_2 = u_1^2+u_2^2+\\cdots+u_n^2\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\begin{cases}\n",
    "\\dfrac{d\\norm{u}_2}{du_1} = 2u_1\\\\\\\\\n",
    "\\dfrac{d\\norm{u}_2}{du_2} = 2u_2\\\\\\\\\n",
    "\\cdots\\\\\\\\\n",
    "\\dfrac{d\\norm{u}_2}{du_n} = 2u_n\n",
    "\\end{cases}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Derivative of the $L^2$ norm\n",
    "\n",
    "In the case of the $L^2$ norm, the derivative is more complicated and takes every elements of the vector into account:\n",
    "\n",
    "$$\n",
    "\\norm{u}_2 = \\sqrt{(u_1^2+u_2^2+\\cdots+u_n^2)} = (u_1^2+u_2^2+\\cdots+u_n^2)^{\\frac{1}{2}}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\begin{align*}\n",
    "\\dfrac{d\\norm{u}_2}{du_1} &=\n",
    "\\dfrac{1}{2}(u_1^2+u_2^2+\\cdots+u_n^2)^{\\frac{1}{2}-1}\\cdot\n",
    "\\dfrac{d}{du_1}(u_1^2+u_2^2+\\cdots+u_n^2)\\\\\\\\\n",
    "&=\\dfrac{1}{2}(u_1^2+u_2^2+\\cdots+u_n^2)^{-\\frac{1}{2}}\\cdot\n",
    "\\dfrac{d}{du_1}(u_1^2+u_2^2+\\cdots+u_n^2)\\\\\\\\\n",
    "&=\\dfrac{1}{2}\\cdot\\dfrac{1}{(u_1^2+u_2^2+\\cdots+u_n^2)^{\\frac{1}{2}}}\\cdot\n",
    "\\dfrac{d}{du_1}(u_1^2+u_2^2+\\cdots+u_n^2)\\\\\\\\\n",
    "&=\\dfrac{1}{2}\\cdot\\dfrac{1}{(u_1^2+u_2^2+\\cdots+u_n^2)^{\\frac{1}{2}}}\\cdot\n",
    "2\\cdot u_1\\\\\\\\\n",
    "&=\\dfrac{u_1}{\\sqrt{(u_1^2+u_2^2+\\cdots+u_n^2)}}\\\\\\\\\n",
    "\\end{align*}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\begin{cases}\n",
    "\\dfrac{d\\norm{u}_2}{du_1} = \\dfrac{u_1}{\\sqrt{(u_1^2+u_2^2+\\cdots+u_n^2)}}\\\\\\\\\n",
    "\\dfrac{d\\norm{u}_2}{du_2} = \\dfrac{u_2}{\\sqrt{(u_1^2+u_2^2+\\cdots+u_n^2)}}\\\\\\\\\n",
    "\\cdots\\\\\\\\\n",
    "\\dfrac{d\\norm{u}_2}{du_n} = \\dfrac{u_n}{\\sqrt{(u_1^2+u_2^2+\\cdots+u_n^2)}}\\\\\\\\\n",
    "\\end{cases}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "One problem of the squared $L^2$ norm is that it hardly discriminates between 0 and small values because the increase of the function is slow.\n",
    "\n",
    "We can see this by graphically comparing the squared $L^2$ norm with the $L^2$ norm. The $z$-axis corresponds to the norm and the $x$- and $y$-axis correspond to two parameters. The same thing is true with more than 2 dimensions but it would be hard to visualize it.\n",
    "\n",
    "$L^2$ norm:\n",
    "\n",
    "<img src=\"images/l2-norm.png\" width=\"500\" alt=\"Representation of the L2 norm\" title=\"The L2 norm\">\n",
    "<em>The L2 norm</em>\n",
    "\n",
    "Squared $L^2$ norm:\n",
    "\n",
    "<img src=\"images/squared-l2-norm.png\" width=\"500\" alt=\"Representation of the squared L2 norm\" title=\"The squared L2 norm\">\n",
    "<em>The squared L2 norm</em>\n",
    "\n",
    "$L^1$ norm:\n",
    "\n",
    "<img src=\"images/L1-norm.png\" alt=\"Representation of the L1 norm\" title=\"The L1 norm\" width=\"500\">\n",
    "<em>The L1 norm</em>\n",
    "\n",
    "These plots are done with the help of this [website](https://academo.org/demos/3d-surface-plotter/). Go and plot these norms if you need to move them in order to catch their shape."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The max norm\n",
    "\n",
    "It is the $L^\\infty$ norm and corresponds to the absolute value of the greatest element of the vector.\n",
    "\n",
    "$$\n",
    "\\norm{\\bs{x}}_\\infty = \\max\\limits_i|x_i|\n",
    "$$\n",
    "\n",
    "# Matrix norms: the Frobenius norm\n",
    "\n",
    "$$\n",
    "\\norm{\\bs{A}}_F=\\sqrt{\\sum_{i,j}A^2_{i,j}}\n",
    "$$\n",
    "\n",
    "This is equivalent to take the $L^2$ norm of the matrix after flattening.\n",
    "\n",
    "The same Numpy function can be use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 2],\n",
       "       [6, 4],\n",
       "       [3, 2]])"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A = np.array([[1, 2], [6, 4], [3, 2]])\n",
    "A"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8.3666002653407556"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.linalg.norm(A)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Expression of the dot product with norms\n",
    "\n",
    "$$\n",
    "\\bs{x}^\\text{T}\\bs{y} = \\norm{\\bs{x}}_2\\cdot\\norm{\\bs{y}}_2\\cos\\theta\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 4.\n",
    "\n",
    "$$\n",
    "\\bs{x}=\n",
    "\\begin{bmatrix}\n",
    "    0 \\\\\\\\\n",
    "    2\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "and \n",
    "\n",
    "$$\n",
    "\\bs{y}=\n",
    "\\begin{bmatrix}\n",
    "    2 \\\\\\\\\n",
    "    2\n",
    "\\end{bmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQIAAAECCAYAAAAVT9lQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEL9JREFUeJzt3XuQVNWBx/HvkGFGN4uQcjCL8YEJcIJJFNdYRNcY40oI\nriZaJJCEMQUGn3kYZyUGEaO1lUfVJsbdTFSCRNYJGiPRhZTIU2IlrEISBcdHnYiYgMuosJnhIc4g\nTO8fPUzQiI3dPX1vd38/VRTdQ9/pXzXTvzn3nnv71GQyGSRVt35JB5CUPItAkkUgySKQhEUgCYtA\nElBbyMYhhMeAjp67z8cYv1R4JEmllncRhBDqgUyM8awi5pGUgEJGBCcC7wwhLAHeAcyIMa4uTixJ\npVTIMYJdwL/HGMcClwPzQggec5DKUCFv3D8C8wBijM8C/wcMKUYoSaVVyK7BRcCHgC+HEI4EBgBt\nB3pwJpPJ1NTUFPB0kvKU841Xk+9FRyGE/sAdwLFAN3BNjPHRt9gks2XLjryeq68MHjyANGVqabmd\npqarUpUJ0vc67ZPGXCnNlLMI8h4RxBhfAxrz3V5SenhwT5JFIMkikIRFIAmLQBIWgSQsAklYBJKw\nCCRhEUjCIpCERSAJi0ASFoEkLAJJWASSsAgkYRFIwiKQhEUgCYtAEhaBJCwCSVgEkihsyTNCCEcA\nvwfOjjH+sTiRJJVa3iOCEEItcBvZVZEllbFCdg2+D9wKbC5SFkkJyasIQgiTgZdjjMs4iJVWJaVb\nXqshhxAeJrsCMsAoIAKfijG+/Bab5bfssqRC9d2y6PuEEFYClx7EwUKXRc/BZdHfnjTmSmmmnEVQ\njOlDf9NLZa6g6UOAGONZxQgiKTmeUCTJIpBkEUjCIpCERSAJi0ASFoEkLAJJWASSsAgkYRFIwiKQ\nhEUgCYtAEhaBJCwCSVgEkrAIJGERSMIikIRFIAmLQBIWgSQsAkkUsMBJCKEfMBsIZNdBvCzG+HSx\ngkkqnUJGBOcBmRjj6cBM4DvFiSSp1PIughjjAuCSnrtDgfZiBJJUegWtfRhj7A4hzAXOBz5TlESS\nSq7gZdEBQghHAGuAkTHGVw/wMFdNlpKRc1n0Qg4WNgJHxRi/B3QCe3v+HFAK141PVaaWlttparoq\nVZkgfa/TPmnMldZMuRSya3AfcEcI4eGe73NljHF3Ad9PUkLyLoIY4y5gYhGzSEqIJxRJsggkWQSS\nsAgkYRFIwiKQhEUgCYtAEhaBJCwCSVgEkrAIJGERSMIikIRFIAmLQBIWgSQsAklYBJKwCCRhEUjC\nIpCERSAJi0ASeS5wEkKoBX5KdhXkOuDbMcZfFTGXpBLKd0TQCGyNMZ4BnAM0Fy+SpFLLd8mzXwD3\n9tyuAV4rThxJScirCHrWPSSEMIBsIcwoZihJpVWTyWTy2jCEcDTZFZGbY4z/dRCb5PdEVaSrq4tN\nmzYxbNiwpKOkVnd3Nw888AAjR470dTp4NbkekO/BwncDS4AvxxhXHux2aVw3Pk2ZrrtuGkOHHsvU\nqV9JOsrrpOF1euWVV/j5z+cxe/atDBs2nJaWewB/pg7G4MEDcj4m32ME04FBwMwQwvVkf9uPizF2\n5fn9BDz11FO0tj6RuiJI0ubN/8ucOT+hpeUOOjo6eOc7/55f/vJX1NTk/CWntyHfYwRfB75e5CxV\nLZPJ8PTTT9HR0cHWrVtpaGhIOlKi2to2c+ONM1m48H727NnT+/XrrruB97znqASTVSZPKEqJZ555\nmvb2djKZDMuXL0k6TuKGDDmSSZO+SG3tX39XnXLKaKZMmZpgqsplEaTE0qUP7nd7cYJJ0qGtbTM3\n3HAdnZ2d1NXVUVdXxw9/2Ey/fv7I9oV8jxGoyPZ/869cuYKuri7q6+sTTJSc1tYnaGycQFvbZq6+\n+ptkMhlqa2sZMSIkHa1iWQQpsGXLFh5//A/U1tbS3d3Nnj2v8cgjqzjzzLOSjlZyy5Yt5uKLp/Da\na7tpbp7FhAmfZ8OG9Rx11DFJR6tojrNSoL39Lyxf/huGDj2OAQMGsGbNOg499O+SjlVyc+bM4sIL\nP0d9fR3z5y9kwoTPA/De9w6jrq4u4XSVzRFBCrxxyDtkyJEMGXJkQmlKb+/evVx//XRmz76N4457\nL3fddS/ve9/wpGNVFYtAidq5cyeXXXYRS5cuZvToU5k79y4OP/zwpGNVHXcNlJi2ts18+tPjWLp0\nMePHT2D+/IWWQEIsAiWitfUJPvnJs2htXcfVV3+TW26ZXbWzJGngroFK7s1mBpQsRwQqqQPNDChZ\njghUEs4MpJtFoD7nzED6uWtQQl1dneT7QTDlypmB8mARlNDKlSt44IEFSccoGWcGyoe7Bn3gnnvm\nsWbNat742Rk7duzgueee5dRTT+fwwyv78wacGSgvFkEfmDhxEhMnTnrd1zo7O/nqVy/lu9/9QcWX\nwJw5s5gx4xoGDhzI3XfP59RT/ynpSMrBIiiRmpoaZs68kWOOGZp0lD7jzED5sghKpL6+vqJLwJmB\n8mYR9IEFC+5j27YONm78M2PHnsNLL7XR3t7Ohg3PccUVX2Pw4COSjlhUbW2baWycSGvrOsaPn8DN\nN//Yg4JlxlmDIlu48H6GDRvBF794EePHT2DmzGsYOHAQ73//SFasWMqGDc8lHbGonBmoDI4Iimzb\ntm184AMfBODFF9vo1+8dfPSjZ9LV1cmPfjSLE088KeGExePMQOVwRFBkF144uff2unWP977x6+sP\nqagS8JqBymIR9KHHHvs9J510ctIximrv3r3MmPENpk+fxrHHDmXRouVOD1aAgooghDA6hHDQS55V\nuu7ubn73u9VkMhna2//C889veF0RzJt3MEtEptfOnTuZPPkLzJ59G6NHn8qiRSucHqwQeRdBCGEa\nMBvwyFCPBQvuo6npK2zatJGHHlrGIYccwhFHZGcIfvObX3Pcce9LOGH+9l0zsGTJg14zUIEKOVi4\nHrgAaClSlrL3wQ+ewJgxn+Shh5YxbNhwpk27lltu+c/eDyMdO/acpCPmZe3atZxzzr/0rjMwbdp0\n1x6sMHkXQYzx/hDCscUMU+6GDx/B9df/2+u+9olPjEsoTXEsW7aYSy6Zwu7dzgxUspJOHx7M8syl\nlqZMxxxzNFu2vJyaTM3NzVx55ZUMGjSIRYsWccYZZyQd6W+k5bXaXxoz5VKMIjjoMWIa141PU6aN\nGzcByb9Ob7xmYPHiB3nXu4aUJFdXVyd1dfUHteuRtv8/SG+mXIoxfVhdn7RR4d5sZmDEiBEle/5q\n+8yGtChoRBBj/DNwWpGyKGGlvGbAz2xIF08xFvC3KxD39cxAtX9mQ9pYBErNNQPV8JkNaeUpxlUu\nTdcMVPpnNqSZRVClkr5m4NFH/4e5c2+nqekrbN++rffrS5cuZvr0q0uWQ1kWQRVK+pqB7du38fzz\nG5g8eSovvfQia9c+3vtvK1dmT81WaVkEVSYN1wysWfMoY8aMZf36Z3nhhU0cf/wHev/tiSfWVtTl\n2uXCIqgiafk0obPPHktDw2AWLVrIySefQkPDYADWr3+WHTt2MGrUP5Y8U7WzCKrEsmWLOe+8sWzd\nuoXm5ll84xvXJn7h0K9//RBnnvnPvffXrXuMww4byNChxyWYqjpZBFUgTTMD+2zfvp0tW15m5Mi/\n7hasXfs4J5wwKsFU1csiqGBJzwy8lf79+9O/f//eUcnGjX9i9epHGDXK4wNJ8ISiCrVz504uv/xL\nLFnyYCrXGTj00EOZNu1afvazuQwfPoIXXniBzs5XOeEEiyAJFkEFKpd1BsaNO5dx484FsrsvAwcO\nIoT3J5yqOrlrUGHSMjOQy+zZt/LII78FIJPJsGLFUsaPn5D4AcxqZRFUkDTODLyZjo4O7rrrTjo6\nOgC4++4WGhoG09g4OdlgVcwiqBBpnBk4kEGDBnHFFVfS3v4XmptvZteuXdx0UzO1te6pJsVXvszt\n3buXb33rWn7yk1vLagXiz372c0lH0H4sgjKW9pkBlQ93DcpUGq4ZUOWwCMrQk0+2lsXMgMqHuwZl\nZvnyJVx88RR27+5ynQEVjSOCMjJnziwaGydSV9c/9TMDKi+OCMpAuc4MqHxYBCnnzIBKwV2DFHNm\nQKWS94gghFAD3AKcCHQCU2OMG4oVrNo9+WQrkyZ91hWIVRKFjAjOB+pjjKcB04GbihNJy5cvKYtr\nBlQ5CimC04HFADHG1cCHi5KoynV1dTkzoJIr5GDhYcC2/e7vCSH0izF2H2iDlpbbC3i64mtquipV\nmXbteoXOzk4aGhqYOnUq69c/w/r1zyQdK3Wv0z5pzJXWTLnUZDL5LWYcQvgB8EiMcX7P/Y0xxmPe\nYhNXTc5h9+7d3HnnnZx//vk0NLj2n4om535lISOCVcC5wPwQwkeA1lwbpHHd+DRlamm5naamq9iy\nZUeqcqXtddonjbnSmimXQorgfmBMCGFVz/0pBXwvSQnKuwhijBng8iJmkZQQTyiSZBFIsggkYRFI\nwiKQhEUgCYtAEhaBJCwCSVgEkrAIJGERSMIikIRFIAmLQBIWgSQsAklYBJKwCCRhEUjCIpCERSAJ\ni0ASFoEkCiyCEMIFIYR5xQojKRl5r3QUQrgZ+ASwtnhxJCWhkBHBKlzyTKoIOUcEIYSLgKvILmte\n0/P3lBjjvSGEj/VxPkklUJPJZPLeuKcILo0xfuEgHp7/E0kqRE2uBxSyLPrblsZ149OUqaXldpqa\nrkpVJkjf67RPGnOlNVMuTh9KKmxEEGN8GHi4SFkkJcQRgSSLQJJFIAmLQBIWgSQsAklYBJKwCCRh\nEUjCIpCERSAJi0ASFoEkLAJJWASSsAgkYRFIwiKQhEUgCYtAEhaBJCwCSVgEkshzXYMQwmHAz4DD\ngP7Av8YYHy1mMEmlk++IoAlYHmM8E5gC/LhoiSSVXL4rHd0EdPXc7g+8Wpw4kpJQyLLofwgh/APQ\nAnytT1NK6lM5iyDG+FPgp2/8egjhQ8BdZI8P/LYPskkqkZpMJvO2NwohHA/8EpgQY2wteipJJZVv\nEfw3cALwJ7K7Cx0xxguKG01SqeRVBJIqiycUSbIIJFkEkrAIJJH/mYVvS9qvTQghXAB8JsY4KcEM\nNcAtwIlAJzA1xrghqTz7CyGMBr4XY/x4CrLUkj2vZShQB3w7xvirhDP1A2YDAegGLosxPp1kpn1C\nCEcAvwfOjjH+8UCPK9WIILXXJoQQbga+TXYaNEnnA/UxxtOA6WRP405cCGEa2R/y+qSz9GgEtsYY\nzwDOAZoTzgNwHpCJMZ4OzAS+k3AeoLc0bwN25XpsqYrgJmBWz+20XZuwCrg86RDA6cBigBjjauDD\nycbptR5I0zkivyD7ZoNseb+WYBYAYowLgEt67g4F2pNL8zrfB24FNud6YNF3DdJ6bcJb5Lo3hPCx\nUud5E4cB2/a7vyeE0C/G2J1UIIAY4/0hhGOTzLC/GOMugBDCAOBeYEayibJijN0hhLlkR3afSTgO\nIYTJwMsxxmUhhGtzPb7oRZDWaxMOlCtFtgMD9rufeAmkVQjhaOA+oDnGeE/SefaJMU7u2SdfE0IY\nGWNMcuQ7BegOIYwBRgF3hhA+FWN8+c0eXKqDhceTHdJ5bcKBrQLOBeaHED4CpO11SvoYCgAhhHcD\nS4AvxxhXJp0HIITQCBwVY/we2QO9e3v+JCbG2DvKDSGsBC49UAlAiYqA7MGTeuA/eo6Oe23C37of\nGBNCWNVzf0qSYd5EWs5Fnw4MAmaGEK4nm2tcjLHrrTfrU/cBd4QQHib7nroyxrg7wTxvlPP/zmsN\nJHlCkSSLQBIWgSQsAklYBJKwCCRhEUjCIpAE/D/dzitW9t/a5QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x108f452d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = [0,0,0,2]\n",
    "y = [0,0,2,2]\n",
    "\n",
    "plt.xlim(-2, 4)\n",
    "plt.ylim(-2, 5)\n",
    "plt.axvline(x=0, color='grey', zorder=0)\n",
    "plt.axhline(y=0, color='grey', zorder=0)\n",
    "\n",
    "plt.quiver([x[0], y[0]],\n",
    "           [x[1], y[1]],\n",
    "           [x[2], y[2]],\n",
    "           [x[3], y[3]],\n",
    "           angles='xy', scale_units='xy', scale=1)\n",
    "\n",
    "plt.text(-0.5, 1, r'$\\vec{x}$', size=18)\n",
    "plt.text(1.5, 0.5, r'$\\vec{y}$', size=18)\n",
    "\n",
    "plt.show()\n",
    "plt.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We took this example for its simplicity. As we can see, the angle $\\theta$ is equal to 45°."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\bs{x^\\text{T}y}=\n",
    "\\begin{bmatrix}\n",
    "    0 & 2\n",
    "\\end{bmatrix} \\cdot\n",
    "\\begin{bmatrix}\n",
    "    2 \\\\\\\\\n",
    "    2\n",
    "\\end{bmatrix} =\n",
    "0\\times2+2\\times2 = 4\n",
    "$$\n",
    "\n",
    "and\n",
    "\n",
    "$$\n",
    "\\norm{\\bs{x}}_2=\\sqrt{0^2+2^2}=\\sqrt{4}=2\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\norm{\\bs{y}}_2=\\sqrt{2^2+2^2}=\\sqrt{8}\n",
    "$$\n",
    "\n",
    "$$\n",
    "2\\times\\sqrt{8}\\times cos(45)=4\n",
    "$$\n",
    "\n",
    "Here are the operations using numpy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4.0000000000000009"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Note: np.cos take the angle in radian\n",
    "np.cos(np.deg2rad(45))*2*np.sqrt(8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span class='notes'>\n",
    "    Feel free to drop me an email or a comment. The syllabus of this series can be found [in the introduction post](https://hadrienj.github.io/posts/Deep-Learning-Book-Series-Introduction/). All the notebooks can be found on [Github](https://github.com/hadrienj/deepLearningBook-Notes).\n",
    "</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# References\n",
    "\n",
    "- https://en.wikipedia.org/wiki/Norm_(mathematics)\n",
    "\n",
    "- [3D plots](https://academo.org/demos/3d-surface-plotter/)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
